Sample-Efficient Integration of New Modalities into LLMs
Table of Contents
When we talk about “integrating” new modalities into LLMs, the common approach is to train an encoder (projector) and insert modality tokens into cross attention layers, as in VLM, or append to textual tokens. These approaches require massive training data, and thus is computational heavy.
Fast adaptation asks whether we can integrate other modalities while leveraging the power of LLM with as little computation as possible. Moreover, fast adaptation wants to handle the problem of variable-sized input.
1. Sample-Efficient Modality Integration (SEMI)1
This paper proposed SEMI, a three-stage integration. The method aims to utilize pretrained modality encoders.
1.1. Train a Shared MLP Projector
Suppose we have \(M\) encoders \(\mathrm{enc}_{i}, i\in[1,M]\) for high-resource2 modalities, and a decoder LLM (hereafter, decoder).
First, we train a projector
\[ \mathrm{proj}_{\phi}(\cdot):\mathbb{R}^{h_{e}} \mapsto \mathbb{R}^{h_{d}} \]
on paired raw data from high-resource modalities. The projector is parameterized by \(\phi\), with \(h_{e}\) being encoder output dimension and \(h_{d}\) being decoder input dimension.
1.2. Train the HyperNetwork
The job of the hypernetwork is to generate a modality-specific adapter, to be composed with the shared projector given only a few data samples.
Suppose for a modality encoder \(\mathrm{enc}_{m}\), an instruction \(\mathbf{i}_{m} \in \mathcal{I}_{m}\) from the instruction pool of that modality, and a sample of examples \( \{ \mathbf{x}_{m},\mathbf{y}_{m} \}_{1}^{S} \in \mathcal{D}_{m} \). The hypernetwork accepts the interleaved encodings as input:
\[ \left( \mathrm{enc}_{\text{text}}(\mathbf{i}_{m}) \oplus \left[ \mathrm{enc}_{m}(\mathbf{x}_{m})\oplus \mathrm{enc}_{\text{text}}(\mathbf{y}_{m}) \right]_{1}^{S} \right) \]
where \(\oplus\) denotes concatenation, and generates LoRA adapters \(\delta\).
Then, the LoRA adapter is plugged into the projector, yielding \( \mathrm{proj}_{\phi+\delta}(\mathbf{x}_{m}) \). The plugged projector, along with the instruction, is fed into the decoder to train the hypernetwork.
Emulating encoders
Since there’re not many modality encoders, this hypernetwork might overfit on a few encoders. Thus, the paper uses random orthogonal matrices to emulate new encoders, which is sampled from \(O(d_{h})\) Haar distribution, where \(d_{h}\) denotes the hypernetwork dimension.
Text grounding
The hypernetwork is instructed with both text embeddings (label and instruction) and modality embeddings, to overcome the issue that different encoders learn different features, and to make text an anchor.
1.3. Few-Shot Adaptation to New Modalities
During adaptation, the hypernetwork, the LLM decoder, and the modality encoder is frozen. The training data is partitioned into batches, for each of which, we generate an adapter. At the end, adapters are averaged and then merged with the pretrained projector. Finally, this updated projector is finetuned on the low-resource modality’s few-shot samples.
1.4. Integrating Arbitrary Dimensionality Encoders
For encoders with smaller output dimensions than the projector input, the pre-trained projector weights are pruned to match the dimension.
For encoders with larger output dimensions than the projector input, the paper uses Infinite Feature Selection (Inf-FS) to reduce the encoder output dimensions.
2. Experiment
Footnotes:
i.e., modalities that have more training resources, e.g., image, audio, video.